Better together: A multistakeholder approach to developing specialty-wide entrustable professional activities in emergency medicine.

Purpose: Entrustable professional activities (EPAs) are a widely used framework for curriculum and assessment, yet the variability in emergency medicine (EM) training programs mandates the development of EPAs that meet the needs of the specialty as a whole. This requires eliciting and incorporating the perspectives of multiple stakeholders (i.e., faculty, residents, and patients) in the development of EPAs. Without a shared understanding of what a resident must be able to do upon graduation, we run the risk of advancing ill-prepared residents that may provide inconsistent care. Methods: In an effort to address these challenges, beginning in February 2020, the authors assembled an advisory board of 25 EM faculty to draft and reach consensus on a final list of EPAs that can be used across all training programs within the specialty of EM. Using modified Delphi methodology, the authors came to consensus on an initial list of 22 EPAs. The authors presented these EPAs to faculty supervisors, residents, and patients for refinement. The authors collated and analyzed feedback from focus groups of residents and patients using thematic analysis. The EPAs were subsequently refined based on this feedback. Results: Stakeholders in EM residency training endorsed a final revised list of 22 EPAs. Stakeholder focus groups highlighted two main thematic considerations that helped shape the finalized list of EM EPAs: attention to the meaningful nuances of EPA language and contextualizing the EPAs and viewing them developmentally. Conclusions: To foreground all key stakeholders within the EPA process for EM, the authors chose within the development process to draft; come to consensus; and refine EPAs for EM in collaboration with relevant faculty, patient, and resident stakeholders. Each stakeholder group contributed meaningfully to the content and intended implementation of the EPAs. This process may serve as a model for others in developing stakeholder-responsive EPAs.

The Next Era of Assessment: Building a Trustworthy Assessment System.

Assessment in medical education has evolved through a sequence of eras each centering on distinct views and values. These eras include measurement (e.g., knowledge exams, objective structured clinical examinations), then judgments (e.g., workplace-based assessments, entrustable professional activities), and most recently systems or programmatic assessment, where over time multiple types and sources of data are collected and combined by competency committees to ensure individual learners are ready to progress to the next stage in their training. Significantly less attention has been paid to the social context of assessment, which has led to an overall erosion of trust in assessment by a variety of stakeholders including learners and frontline assessors. To meaningfully move forward, the authors assert that the reestablishment of trust should be foundational to the next era of assessment. In our actions and interventions, it is imperative that medical education leaders address and build trust in assessment at a systems level. To that end, the authors first review tenets on the social contextualization of assessment and its linkage to trust and discuss consequences should the current state of low trust continue. The authors then posit that trusting and trustworthy relationships can exist at individual as well as organizational and systems levels. Finally, the authors propose a framework to build trust at multiple levels in a future assessment system; one that invites and supports professional and human growth and has the potential to position assessment as a fundamental component of renegotiating the social contract between medical education and the health of the public.

The Inconspicuous Learner Handover: An Exploratory Study of U.S. Emergency Medicine Program Directors' Perceptions of Learner Handovers from Medical School to Residency.

Phenomenon: Central to competency-based medical education is the need for a seamless developmental continuum of training and practice. Trainees currently experience significant discontinuity in the transition from undergraduate (UME) to graduate medical education (GME). The learner handover is intended to smooth this transition, but little is known about how well this is working from the GME perspective. In an attempt to gather preliminary evidence, this study explores U.S. program directors (PDs) perspective of the learner handover from UME to GME. Approach: Using exploratory qualitative methodology, we conducted semi-structured interviews with 12 Emergency Medicine PDs within the U.S. from October to November, 2020. We asked participants to describe their current perception of the learner handover from UME to GME. Then we performed thematic analysis using an inductive approach. Findings: We identified two main themes: The inconspicuous learner handover and barrier to creating a successful UME to GME learner handover. PDs described the current state of the learner handover as "nonexistent," yet acknowledged that information is transmitted from UME to GME. Participants also highlighted key challenges preventing a successful learner handover from UME to GME. These included: conflicting expectations, issues of trust and transparency, and a dearth of assessment data to actually hand over. Insights: PDs highlight the inconspicuous nature of learner handovers, suggesting that assessment information is not shared in the way it should be in the transition from UME to GME. Challenges with the learner handover demonstrate a lack of trust, transparency, and explicit communication between UME and GME. Our findings can inform how national organizations establish a unified approach to transmitting growth-oriented assessment data and formalize transparent learner handovers from UME to GME.

Lessons From Organic Chemistry: The Case for Considering Both High Standards and Equity in Assessment.

ABSTRACT: In this commentary, the authors explore the tension of balancing high performance standards in medical education with the acceptability of those standards to stakeholders (e.g., learners and patients). The authors then offer a lens through which this tension might be considered and ways forward that focus on both patient outcomes and learner needs.In examining this phenomenon, the authors argue that high performance standards are often necessary. Societal accountability is key to medical education, with the public demanding that training programs prepare physicians to provide high-quality care. Medical schools and residency programs, therefore, require rigorous standards to ensure graduates are ready to care for patients. At the same time, learners' experience is important to consider. Making sure that performance standards are acceptable to stakeholders supports the validity of assessment decisions.Equity should also be central to program evaluation and validity arguments when considering performance standards. Currently, learners across the continuum are variably prepared for the next phase in training and often face inequities in resource availability to meet high passing standards, which may lead to learner attrition. Many students who face these inequities come from underrepresented or disadvantaged backgrounds and are essential to ensuring a diverse medical workforce to meet the needs of patients and society. When these students struggle, it contributes to the leaky pipeline of more socioeconomically and racially diverse applicants.The authors posit that 4 key factors can balance the tension between high performance standards and stakeholder acceptability: standards that are acceptable and defensible, progression that is time variable, requisite support structures that are uniquely tailored for each learner, and assessment systems that are equitably designed.

Postgraduate Selection in Medical Education: A Scoping Review of Current Priorities and Values.

PURPOSE: The process of screening and selecting trainees for postgraduate training has evolved significantly in recent years, yet remains a daunting task. Postgraduate training directors seek ways to feasibly and defensibly select candidates, which has resulted in an explosion of literature seeking to identify root causes for the problems observed in postgraduate selection and generate viable solutions. The authors therefore conducted a scoping review to analyze the problems and priorities presented within the postgraduate selection literature to explore practical implications and present a research agenda. METHOD: Between May 2021 and February 2022, the authors searched PubMed, EMBASE, Web of Science, ERIC, and Google Scholar for English language literature published after 2000. Articles that described postgraduate selection were eligible for inclusion. 2,273 articles were ultimately eligible for inclusion. Thematic analysis was performed on a subset of 100 articles examining priorities and problems within postgraduate selection. Articles were sampled to ensure broad thematic and geographical variation across the breadth of articles that were eligible for inclusion. RESULTS: Five distinct perspectives or value statements were identified in the thematic analysis: (1) Using available metrics to predict performance in postgraduate training; (2) identifying the best applicants via competitive comparison; (3) seeking alignment between applicant and program in the selection process; (4) ensuring diversity, mitigation of bias, and equity in the selection process; and (5) optimizing the logistics or mechanics of the selection process. CONCLUSIONS: This review provides insight into the framing and value statements authors use to describe postgraduate selection within the literature. The identified value statements provide a window into the assumptions and subsequent implications of viewing postgraduate selection through each of these lenses. Future research must consider the outcomes and consequences of the value statement chosen and the impact on current and future approaches to postgraduate selection.

TRainee Attributable & Automatable Care Evaluations in Real-time (TRACERs): A Scalable Approach for Linking Education to Patient Care.

Competency-based medical education (CBME) is an outcomes-based approach to education and assessment that focuses on what competencies trainees need to learn in order to provide effective patient care. Despite this goal of providing quality patient care, trainees rarely receive measures of their clinical performance. This is problematic because defining a trainee's learning progression requires measuring their clinical performance. Traditional clinical performance measures (CPMs) are often met with skepticism from trainees given their poor individual-level attribution. Resident-sensitive quality measures (RSQMs) are attributable to individuals, but lack the expeditiousness needed to deliver timely feedback and can be difficult to automate at scale across programs. In this eye opener, the authors present a conceptual framework for a new type of measure - TRainee Attributable & Automatable Care Evaluations in Real-time (TRACERs) - attuned to both automation and trainee attribution as the next evolutionary step in linking education to patient care. TRACERs have five defining characteristics: meaningful (for patient care and trainees), attributable (sufficiently to the trainee of interest), automatable (minimal human input once fully implemented), scalable (across electronic health records [EHRs] and training environments), and real-time (amenable to formative educational feedback loops). Ideally, TRACERs optimize all five characteristics to the greatest degree possible. TRACERs are uniquely focused on measures of clinical performance that are captured in the EHR, whether routinely collected or generated using sophisticated analytics, and are intended to complement (not replace) other sources of assessment data. TRACERs have the potential to contribute to a national system of high-density, trainee-attributable, patient-centered outcome measures.

Competency-Based Medical Education in a Norm-Referenced World: A Root Cause Analysis of Challenges to the Competency-Based Paradigm in Medical School.

Competency-based medical education (CBME) requires a criterion-referenced approach to assessment. However, despite best efforts to advance CBME, there remains an implicit, and at times, explicit, demand for norm-referencing, particularly at the junction of undergraduate medical education (UME) and graduate medical education (GME). In this manuscript, the authors perform a root cause analysis to determine the underlying reasons for continued norm-referencing in the context of the movement toward CBME. The root cause analysis consisted of 2 processes: (1) identification of potential causes and effects organized into a fishbone diagram and (2) identification of the 5 whys. The fishbone diagram identified 2 primary drivers: the false notion that measures such as grades are truly objective and the importance of different incentives for different key constituents. From these drivers, the importance of norm-referencing for residency selection was identified as a critical component. Exploration of the 5 whys further detailed the reasons for continuation of norm-referenced grading to facilitate selection, including the need for efficient screening in residency selection, dependence upon rank-order lists, perception that there is a best outcome to the match, lack of trust between residency programs and medical schools, and inadequate resources to support progression of trainees. Based on these findings, the authors argue that the implied purpose of assessment in UME is primarily stratification for residency selection. Because stratification requires comparison, a norm-referenced approach is needed. To advance CBME, the authors recommend reconsideration of the approach to assessment in UME to maintain the purpose of selection while also advancing the purpose of rendering a competency decision. Changing the approach will require a collaboration between national organizations, accrediting bodies, GME programs, UME programs, students, and patients/societies. Details are provided regarding the specific approaches required of each key constituent group.

Bridging the gap: Development of an experiential learning-based health disparities curriculum.

Background: The increasing number of vulnerable populations served by the emergency department (ED) calls for developing and implementing curricula aimed at training residents to deliver quality care for the most marginalized groups. Objective: We aimed to address this by developing and piloting a curriculum to introduce the unique challenges and disparities encountered by our diverse ED patient population using an experiential learning approach. Methods: We engaged community partners in designing and implementing a curriculum for incoming interns. This curriculum addresses specific populations encountered within the ED including patients with psychiatric illness, patients with cognitive disabilities, intoxicated patients, violent patients, patients of various cultural backgrounds, non-English-speaking patients, and LGBTQ patients. Experiential and active learning sessions with content experts and site visits to area organizations were arranged. Pre-, post-, and time-delayed surveys were deployed to evaluate the pilot of this curriculum. Results: Thirteen incoming interns participated in the orientation curriculum. Residents' comfort with each of these various patient populations as well as familiarity with community and ED resources was assessed before, after, and 1 year delayed from each session (response 13/13, 100%). Their scores increased significantly from baseline in the postsurvey and were maintained 1 year later (p < 0.05). Residents additionally provided narrative responses regarding what they learned and what was most helpful after completing their intern year. Conclusions: This pilot curriculum demonstrates that implementing an experiential learning curriculum and engaging community leaders and resources is key to training residents to address health disparities within their unique ED patient population. As such, it is imperative that we seek to immersively introduce trainees to the unique needs of the patient population they will serve early in training.

The Purpose, Design, and Promise of Medical Education Research Labs.

Medical education researchers are often subject to challenges that include lack of funding, collaborators, study subjects, and departmental support. The construct of a research lab provides a framework that can be employed to overcome these challenges and effectively support the work of medical education researchers; however, labs are relatively uncommon in the medical education field. Using case examples, the authors describe the organization and mission of medical education research labs contrasted with those of larger research team configurations, such as research centers, collaboratives, and networks. They discuss several key elements of education research labs: the importance of lab identity, the signaling effect of a lab designation, required infrastructure, and the training mission of a lab. The need for medical education researchers to be visionary and strategic when designing their labs is emphasized, start-up considerations and the likelihood of support for medical education labs is considered, and the degree to which department leaders should support such labs is questioned.

A mixed-methods needs assessment to identify pharmacology education objectives for emergency medicine residents.

Objectives: Medication errors represent a significant threat to patient safety. Pharmacotherapy is one of the 23 Accreditation Council of Graduate Medical Education milestones for emergency medicine, yet there is minimal understanding of what content should be prioritized during training. The study aim was to develop objectives for a patient-safety focused pharmacology curriculum for emergency medicine residents. Methods: We incorporated data from a de-identified safety event database and survey responses of 30 faculty and clinical pharmacists at a single-site suburban university hospital with 24-hour emergency medicine pharmacists and an annual volume of approximately 70,000. We reviewed the database to quantify types and severity of medication errors over a 5-year period for a total of 370 errors. Anonymous surveys included categorical items that we analyzed with descriptive statistics and short answer questions that underwent thematic analysis by 2 coders. We summarized all data sources to identify curriculum gaps. Results: Common medication errors reported in our database were wrong dose (43%) and computer order entry errors (14%). Knowledge gaps were medication cost (63%), pregnancy risk information (60%), antibiotic stewardship (53%), interactions (47%), and side effects (47%). Qualitative analysis revealed the need to optimize computer order entry, understand the scope of critical medications, use references, and consult pharmacists. Integration of data suggested specific medications should be covered in curricular efforts, including antibiotics, analgesics, sedatives, and insulin. Conclusion: We developed objectives of pharmacology topics to prioritize during emergency medicine training to enhance prescribing safety. This study is limited due to its small sample and single institution source of data. Future studies should investigate the impact of pharmacology curriculum on minimizing clinical errors.

An Outcomes-Oriented Approach to Residency Selection: Implementing Novel Processes to Align Residency Programs and Applicants.

Residency application numbers have skyrocketed in the last decade, and stakeholders have scrambled to identify and deploy methods of reducing the number of applications submitted to each program. These interventions have traditionally focused on the logistics of the application submission and review process, neglecting many of the drivers of overapplication. Implementing application caps, preference signaling as described by Pletcher and colleagues in this issue, or an early Match does not address the fear of not matching that applicants hold, the lack of transparent data available for applicants to assess their alignment with a specific program, or issues of inequity in the residency selection process. Now is the time to reconsider the residency selection process itself. As competency-based medical education emerges as the predominant educational paradigm, residency selection practices must also shift to align with societal, specialty, and program outcomes. The field of industrial and organizational psychology offers a multitude of tools (e.g., job analysis) by which to define the necessary outcomes of residency training. These tools also provide programs with the infrastructure around which to scaffold an outcomes-oriented approach to the residency selection process. Programs then can connect residency selection to training outcomes, longitudinal assessment modalities, and the evolving learning environment. To achieve an outcomes-oriented residency selection process, stakeholders at all levels will need to invest in coproducing novel ways forward. These solutions range from defining program priorities to implementing national policy. Focusing on outcomes will facilitate a more transparent residency selection process while also allowing logistics-level interventions to be successful, as applicants will be empowered to better assess their alignment with each program and apply accordingly.

Development, Implementation, and Evaluation of an Open Access, Level-Specific, Core Content Curriculum for Emergency Medicine Residents.

BACKGROUND: Leaders in graduate medical education must provide robust clinical and didactic experiences to prepare residents for independent practice. Programs traditionally create didactic experiences individually, requiring tremendous resources with variable content exposure and quality. OBJECTIVE: We sought to create and implement a free, open access, learner-centric, level-specific, emergency medicine (EM) residency curriculum. METHODS: We developed Foundations of Emergency Medicine (FoEM) Foundations I and II courses using Kern's model of curriculum development. Fundamental topics were identified through content guidelines from the American Board of Emergency Medicine. We incorporated learner-centric strategies into 2 flipped classroom, case-based courses targeting postgraduate year (PGY) 1 and PGY-2 residents. The curriculum was made freely available online in 2016. Faculty and resident users were surveyed annually for feedback, which informed iterative refinement of the curriculum. RESULTS: Between 2016 and 2020, registration for FoEM expanded from 2 sites with 36 learners to 154 sites and 4453 learners. In 2019, 98 of 102 (96%) site leaders and 1618 of 2996 (54%) learners completed the evaluative survey. One hundred percent of responding leaders and 93% of learners were "satisfied" or "very satisfied" with FoEM content. Faculty and residents valued FoEM's usability, large volume of content, quality, adaptability, organization, resident-faculty interaction, and resident-as-teacher opportunities. Challenges to implementation included resident attendance, conference structure, technology limitations, and faculty engagement. CONCLUSIONS: We developed and implemented a learner-centric, level-specific, national EM curriculum that has been widely adopted in the United States.

Assessment of Entrustable Professional Activities Using a Web-Based Simulation Platform During Transition to Emergency Medicine Residency: Mixed Methods Pilot Study.

BACKGROUND: The 13 core entrustable professional activities (EPAs) are key competency-based learning outcomes in the transition from undergraduate to graduate medical education in the United States. Five of these EPAs (EPA2: prioritizing differentials, EPA3: recommending and interpreting tests, EPA4: entering orders and prescriptions, EPA5: documenting clinical encounters, and EPA10: recognizing urgent and emergent conditions) are uniquely suited for web-based assessment. OBJECTIVE: In this pilot study, we created cases on a web-based simulation platform for the diagnostic assessment of these EPAs and examined the feasibility and acceptability of the platform. METHODS: Four simulation cases underwent 3 rounds of consensus panels and pilot testing. Incoming emergency medicine interns (N=15) completed all cases. A maximum of 4 "look for" statements, which encompassed specific EPAs, were generated for each participant: (1) performing harmful or missing actions, (2) narrowing differential or wrong final diagnosis, (3) errors in documentation, and (4) lack of recognition and stabilization of urgent diagnoses. Finally, we interviewed a sample of interns (n=5) and residency leadership (n=5) and analyzed the responses using thematic analysis. RESULTS: All participants had at least one missing critical action, and 40% (6/15) of the participants performed at least one harmful action across all 4 cases. The final diagnosis was not included in the differential diagnosis in more than half of the assessments (8/15, 54%). Other errors included selecting incorrect documentation passages (6/15, 40%) and indiscriminately applying oxygen (9/15, 60%). The interview themes included psychological safety of the interface, ability to assess learning, and fidelity of cases. The most valuable feature cited was the ability to place orders in a realistic electronic medical record interface. CONCLUSIONS: This study demonstrates the feasibility and acceptability of a web-based platform for diagnostic assessment of specific EPAs. The approach rapidly identifies potential areas of concern for incoming interns using an asynchronous format, provides feedback in a manner appreciated by residency leadership, and informs individualized learning plans.

Diagnosing conflict: Conflicting data, interpersonal conflict, and conflicts of interest in clinical competency committees.

Clinical competency committees (CCCs) are increasingly used within health professions education as their decisions are thought to be more defensible and fairer than those generated by previous training promotion processes. However, as with most group-based processes, it is inevitable that conflict will arise. In this paper the authors explore three ways conflict may arise within a CCC: (1) conflicting data submissions that are presented to the committee, (2) conflicts between members of the committee, and (3) conflicts of interest between a specific committee member and a trainee. The authors describe each of these conflict situations, dissect out the underlying problems, and explore possible solutions based on the current literature.

If we could turn back time: Imagining time-variable, competency-based medical education in the context of COVID-19.

The COVID-19 pandemic has exposed a paradox in historical models of medical education: organizations responsible for applying consistent standards for progression have needed to adapt to training environments marked by inconsistency and change. Although some institutions have maintained their traditional requirements, others have accelerated their programs to rush nearly graduated trainees to the front lines. One interpretation of the unplanned shortening of the duration of training programs during a crisis is that standards have been lowered. But it is also possible that these trainees were examined according to the same standards as usual and were judged to have already met them. This paper discusses the impacts of the COVID-19 pandemic on the current workforce, provides an analysis of how competency-based medical education (CBME) in the context of the pandemic might have mitigated wide-scale disruption, and identifies structural barriers to achieving an ideal state. The paper further calls upon universities, health centres, governments, certifying bodies, regulatory authorities, and health care professionals to work collectively on a truly time-variable model of CBME. The pandemic has made clear that time variability in medical education already exists and should be adopted widely and formally. If our systems today had used a framework of outcome competencies, sequenced progression, tailored learning, focused instruction, and programmatic assessment, we may have been even more nimble in changing our systems to care for our patients with COVID-19.

Curated Collections for Clinician Educators: Five Key Papers on Graduated Responsibility in Residency Education.

Introduction The Accreditation Council for Graduate Medical Education calls graduated responsibility "one of the core tenets of American graduate medical education." However, there is no clear set of resources for programs to implement a system of progressively increasing responsibilities for trainees. This project aimed to identify a set of high-yield papers on graduated responsibility for junior faculty members. Methods A study group of Academic Life in Emergency Medicine Faculty Incubator participants identified relevant literature on graduated responsibility via a comprehensive literature search and a call to the online medical education community; 59 total papers were identified. The most relevant and applicable were selected by the study group via a three-round modified Delphi process. Results Five key articles for junior faculty interested in implementing more robust graduated responsibility at their residency training program were selected and described here. Summaries of key points, along with considerations for faculty developers and relevance to junior faculty, are presented for each article. Conclusions The articles presented here provide a solid theoretical and practical basis for junior faculty to explore graduated responsibility. The five articles presented here provide the junior faculty with a toolkit to examine and improve their systems for assigning responsibilities in a graded fashion at their own institutions.